Method Y ear Family Train T est V alidation (HP/Model Selection) RandNet [11] 2017 AE ensemble Polluted =Train None, fixed - sensitivity analysis on some HPs RDA [48] 2017 AE Polluted =Train Best on Test, other HPs fixed DAGMM [49] 2018 AE & density Clean & Pol.d Disjoint None, fixed - sensitivity on reg. For MNIST, we choose Digit '4' and '5' For CIFAR10, we choose class'automobile' as the inlier-class. The inliers are labeled 0 and outliers are denoted as label 1. The data split and configuration are the same as described in the authors' provided code. With 4-to-8 different HPs each, the total number of configurations, and i.e. models trained, After the first-layer, the number of channels expand at rate of 2 .

Outlier detection (OD) literature exhibits numerous algorithms as it applies to diverse domains. However, given a new detection task, it is unclear how to choose an algorithm to use, nor how to set its hyperparameter(s) (HPs) in unsupervised settings. HP tuning is an ever-growing problem with the arrival of many new detectors based on deep learning, which usually come with a long list of HPs. Surprisingly, the issue of model selection in the outlier mining literature has been "the elephant in the room"; a significant factor in unlocking the utmost potential of deep methods, yet little said or done to systematically tackle the issue. In the first part of this paper, we conduct the first large-scale analysis on the HP sensitivity of deep OD methods, and through more than 35,000 trained models, quantitatively demonstrate that model selection is inevitable. Next, we design a HP-robust and scalable deep hyper-ensemble model called ROBOD that assembles models with varying HP configurations, bypassing the choice paralysis. Importantly, we introduce novel strategies to speed up ensemble training, such as parameter sharing, batch/simultaneous training, and data subsampling, that allow us to train fewer models with fewer parameters. Extensive experiments on both image and tabular datasets show that ROBOD achieves and retains robust, state-of-the-art detection performance as compared to its modern counterparts, while taking only $2$-$10$\% of the time by the naive hyper-ensemble with independent training.